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Experimentation 


□  Refine  a  nomological  network  regarding 
macrocognition 

■  How  many  distinct  components? 

■  How  are  they  related? 

■  How  are  they  affected  by  contextual  variables? 
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Overarching  Objective 


To  identify  a  set  of  metrics  that  are 
unobtrusive,  construct  valid,  incrementally 
predictive,  and  sensitive  to  our  manipulations 
for  use  in  experimentally-testing  our 
hypotheses. 
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Experimental  Tasks 


□  Validity  of  metrics  should  generalize  across 
tasks  that  share  substantive  features 


□  Two  task  environments  have  been 
developed  and  pilot-tested 

■  NEO  scenario 

■  ER  simulation 
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Our  Philosophy  Regarding  Metrics 
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Process  v.  Product  at  Team  or 
Individual  Level  of  Analysis 


Individual 


Product 

Process 

•  Concept  maps 

•  Point  of  regard  at  key  events 

•  Degree  of  overlap 

•  Complementary  point  of 
regard  at  key  events 

Team 


Components  of  a  Metric 


□  Content:  for  example,  terms,  behaviors, 
physiological  reactions 

□  Source:  for  example,  participant,  peers,  outside 
observers,  equipment 

□  Method:  for  example,  Likert-type  scale, 
checklist,  card-sorting  task,  pairwise 
comparisons 

□  Scoring/indexing:  for  example,  percentage, 
mean,  sum,  correlation  to  expert,  distance 
from  expert 


Content 
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Content 


□  Cognitive:  lots  of  construct  confusion 

□  Physiological:  very  little  done  in  team  arena, 
however  some  preliminary  evidence  that  teams 
with  similar  reactions  perform  better 

□  Attitudinal:  levels  of  analysis  issues 

□  Behavioral:  problems  with  discriminant  validity 


«U4bi»  kr  HaiiMAn  h  indtAg 


ILLINOIS 

uniMf 


Methods 


6 

«U4bi»  kr  HaiiMAn  h  indtAg 


ILLINOIS 


MEDNASrm 

llNiVERSm' 


Major  Methods  for  Measuring  Cognitive  Team 

Constructs 


□  Concept  maps 

□  Pair-wise  comparison  ratings 

□  Card  sorting 

□  Vignette-based  ratings 

□  Think-aloud  protocols 

□  Probes  during  task  performance 


□  NEED  TRIANGULATION 
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Card  Sorting  Task 


h^-forntatioi^ 


ader^ki'p 


At  the  beginning 
the  manager  to 
staff  how  many  < 
the  kitchen  ha 
reservations  the 
there  were  any 
comin 


The  hostess  offered  to  take 
drink  orders  for  the  table  she 
had  just  seated  since  the 
server  had  his  hands  full  with 
a  large  party. 


Behavior 


^r  suggested  that  the 
Id  increase  his  wine 
icked  out  a  few  of  his 
to  recommend  to 
guests. 
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Conciseness  &  Grouping  Strategy 


Server 

Coordination 


Kitchen 

Coordination 


When  the  kitchen 
started  running 
behind  on  orders, 
the  chef  asked  the 
manager  to  jump  in 
to  heip  them  catch 
up. 


Restaurant 

Coordination 


1 

The  manager 
suggested  that  the 
server  couid 
increase  his  wine 
saies  if  he  picked  out 
a  few  of  his  favorites 
to  recommend  to 
guests. 

Conciseness  score: 


Grouping  strategy: 
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Concrete 


Vignette-Based 


Teammate  A  appears  to  you  to  be  overloaded  and  in  need  to 
assistance.  If  you  were  to  offer  assistance  to  teammate  A 
how  likely  would  he/she  be  to: 

Defensively  refuse  your  assistance: 

Politely  refuse  your  assistance: 

Gratefully  accept  your  assistance: 

Grudgingly  accept  your  assistance: 

Embarrassingly  accept  your  assistance: 

Highly  unlikely  Highly  likely 

1- — 2- — 3- — 4- — 5- — 6 


Relatedness  Ratings 


Participant’s  SME’s  Ratings 
Ratings 


To  what  degree  are  the  following  goals  related  and  in  which  direction? 


-2  -1 


Maintaining 

Separation 


Strong  negative 


"  o  © 

Unrelated  Strong  Positive 


Minimizing 

Runway 

Delays 


Maintaining 

Separation 


■2  (0) 

Strong  negative 


0 

Unrelated 


2 


Very  positively 


Minimizing 

go-arounds 


Maintaining 

Separation 


©  ■' 

Strong  negative 


© 

Unrelated 


2 

Strong  positive 


Avoiding 

windsheer 


Major  Methods  for  Measuring 
Behavioral/communication-related  Team 

Constructs 


□  Likert  scales 

□  Frequency  counts 

□  Behaviorally  anchored  rating  scales 
Checklists 

□  Communication  flow/sequencing 


Each  have  strengths  and  weaknesses  -  construct  and  context  must  be 
considered  when  determining  which  to  use! 
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Frequency  Counts 


Number  of  times  that  backup  was 
demonstrated : 


Likert-type  Scale 
Backup  Behavior  Was: 

Highly  Ineffective  Highly  Effective 
1 - 2 - 3 - 4 - 5 - 6 
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Behaviorally  Anchored  Rating  Scale  (BARS) 


TyptcdLy.  in  thla  ojc^aniaatioD .  ooe  expect  oiechanisnia  to  have  been  Introduced  ioi 
the  sxpjesa  purpose  of  cascadiog  infonnation  a^^teoMlioaJLy  £roni  top  b  boHooi  of  the 
oj^nLaationa]  bierarchy . 


i.5 


TypicdLj.  Ln  Ihi s  oieaniaatioD .  one  ^ould  expect  a  Hiana^eiDeot  LnfccjnatioD  pcovisiDD  unit 
to  have  been  Ktabllahed  which  la  constantly  consultLn  e;  infoimatiDn  uaeja  od  theii  pj^eaent 
and  future  needs. 


4  -  -  TyptcilLj.  Id  thla  o^g^£lL£alion .  one  ^'ould  expec t  mlnutea  of  governlo ^  body  jneeling  (e.^ 

Board  of  Goveinoja/Council'' Academic.  Board)  to  be  oisrie  avaibbJe  to  all  alaff,  and 
actively  circulated  to  thoae  ^'bo  need  them. 

3.5  —  TypicdLy.  in  this  ccganisatioD ,  one  would  expec  t  iDfoioiado  n  bulletin  a  from  mana^meot, 
and  meetings,  to  focus  piimariL^  on  deveiopmenta  that  bave  already  taken  place^  aa 
opposed  to  deveiopmenta  In  the  pipeline. 


3 


TypicdLy.  in  this  orcanlsation.  one  would  stpect  tbeie  to  be  both  formal  and  informal 
channels  for  Information  but  information  provision  to  be  not  aJwaya  timelj. 


Typicdly.  in  this  oiganlssidon.  one  would  Mpect  thece  to  be  witfeapread  use  of  the 
'confideotiar  and  "jeSricted'  stamp  on  documents  and  reports. 


TyptcalLj,  In  thla  organisation,  one  would  expect  information  produced  centraJLy  to  be 
consigned  to  the  waste  bin  frequently  by  recipients  because  It  is  thought  to  serve  no  useful 
purpose. 


2  —  Typically,  in  this  or^lsation,  one  would  e?(pect  Information  proviaion  to  be  "ad  hoc'  in 

the  sense  of  being  provided  when  requested  if  one  happens  to  know  that  it  is  available  and 
the  relevant  party  to  contact. 

1.5  -  -  Typicdly.  In  thla  ojgjinbation.  one  would  expect  little  or  no  publicity  to  be  ^ven  to  major 
developments  such  aa  the  setting  up  of  a  new  unit  or  the  Introduction  of  a  new  ifrcility. 


t 


Checklists 


YES 

NO 

Assisted  AAWC  in  identifying  tracks  at  first 
wave. 

X 

Assisted  AAWC  in  identifying  tracks  at 
second  wave. 

X 

Assisted  AAWC  in  identifying  tracks  at  third 
wave. 

X 

Assisted  AAWC  in  identifying  tracks  at  fourth 

X 

wave. 


Flow  Patterns  Change  in 

Real-time 


Before  Conflict  After  Conflict 


Major  Methods  for  Measuring  Physiological 

Team  Constructs 

□  Synchronous: 

■  Eye  scan/fixation 

■  Pupil  dilation 

■  Increases  in  heart  rate/blood  pressure 

■  Vocal  intensity,  pitch 
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Rule  3;  Doctor?  ha\e  the  liglit  to  dem'  access  to  a  patient  if  it  is  in  the  best  interest  of  theujifnaif  s 
medical  conditioii 

Rule  4:  Yon  will  be  fired  if  yon  lie  \'eibally  or  in  wiiting. 

Rule  5;  No  racial  or  sexual  liai-assing  conmients. 


Patient 

Gender 

Consent 

^dical  Condijiim 

Notes 

Name 

Michael 

Rayfield 

Male 

Doctors 

visitors 

Severe  trauma 
from  auto  accident 

Currently  in  the  intensive 
care  unit 

Jane  Doe 

Female 

Non-publicity  patient 

Complications 
during  miscarriage 

Real  name:  Kayla 

Johnson 

Carmen 

Diaz 

Female 

Only  grandfather, 
Mannv  Diaz,  mav 
see  Carmen 

Several  broken 
bones  and 
concussion 

Mother  suspected  of  child 
abuse.  Sire's  not  aibwed 
to  see  the  child. 

□  Intensity  analysis  indicates  the  degree  of  aggressive  or 
assertive  tendencies  between  team  members. 


Participant  A 


. 


Visible  part  9.000000  seconds 


Total  duration  9.000000  seconds 


Participant  B  (More  Aggressive) 


IB 

4.633904 

■1:  II  III 

lli^ 

4.633904 

0.000000  Visible  part 

Total  duratio 

;  100  dB 

pTlI 

67.^2  c 

;  50  dB 

2.366096 

7 .000000  se  CO  n  ds  7 .000000 

n  7.000000  seconds 

□  Pitch  analysis  indicates  the  degree  of  emotion  on  the 
person's  voice  in  a  stressful  scenario. 

Participant  A  (More  emotional)  Participant  B 


m 

1 .000000 

W 

600  Hz 

166.27  Hz 

75  Hz 

.  r~'  ^  “ 

'  Ai  ~ 

1.000000  I  1.000000 

0 .000000  Visiblepart2 .000000  seconds  2 .000000 

Total  duration  2.000000  seconds 

Autonomic  Measures  and  Findings 


Measure 

Related  Construct 

Previous  Findings 

Reference 

Heart  Rate 
(HR) 

Self  monitoring 

HR  activity  during  anticipation  of  socially  threatening  situation  (public  speech) 
was  negatively  related  to  self  monitoring. 

Hofman,  2006 

Self-efficacy 

Positively  associated  with  hr  and  unrelated  to  self  reported  arousal. 

Gellatly  &  Meyer,  1992 

Personal  goal  level 

Positively  associated  with  hr  and  unrelated  to  self  reported  arousal. 

Gellatly  &  Meyer,  1992 

Goal  difficulty 

Positively  associated  with  HR. 

Gellatly  &  Meyer,  1992 

Performance 

HR  positively  associated  with  task  performance  but  unrelated  to  self  reported 
arousal. 

Gellatly  &  Meyer,  1992 

Team  performance 

Team  change  in  HR  similarity  positively  predicted  overall  performance  and  errors 
made  on  a  team  coordination  task  involving  robotic  operation. 

Henning  et  al.,  2001 

Blood  pressure 

Task  difficulty 

Social  Presence 

Blood  pressure  higher  when  being  observed  compared  to  no  observation  while  a 
performing  difficult  task.  This  was  not  found  for  an  easy  task. 

Henning  et  al.,  2001 

Galvanised  Skin 
Response 
(GSR) 

Self  Monitoring 

Social  Threat 

GSR  activity  during  anticipation  of  socially  threatening  situation  (public  speech) 
was  negatively  related  to  self  monitoring. 

Hofman,  2006 

Learning 

GSR  strength  during  training  predicted  learning  in  support  of  Damasio’s  Somatic 
Hypothesis. 

Carter  &  Pasqualini,  2004 

Cortisol 

Stress/Anxiety 

Cortisol  sensitive  to  stress  and  anxiety. 

Schlotz  et  al.,  2006 

Criteria  for  Metrics 
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Criteria  for  Metrics 


□  Construct  validity 

□  Incremental  predictive  validity 

□  Sensitivity 


□  Unobtrusiveness 
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Construct  Validity 


□  To  what  degree  does  a  metric  capture 
the  construct  of  interest? 

□  Convergent  validity:  Should  correlate 
highly  with  other  metrics  of  the  same 
construct  -  TRIANGULATION 

□  Discriminant  validity:  Should  not 
correlate  highly  with  metrics  of 
theoretically  distinct  constructs. 


Sensitivity 


□  To  what  degree  does  the  metric  detect 
differences  among  participants  related  to  other 
variables/manipulations  of  interest? 

□  Floor  and  ceiling  effects 

□  Quantity  versus  quality 


□  Ordinal  versus  interval  or  ratio  scales 
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Unobtrusiveness 


□  To  what  degree  does  administration  of  the 
metric  itself  influence  participants 

□  A  potential  source  of  contamination 

□  "Testing"  threat  to  validity 


□  Can  be  assessed  by  comparing  participants  who 
were  administered  the  metric  (e.g., 
eyetracking)  to  those  who  were  not  on  the 
dependent  variables  of  interest 
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Incremental  Predictive  Validity 


□  To  what  degree  do  multiple  measures  of  macro¬ 
cognition  contribute  uniquely  to  the  prediction  of 
team  performance  (explain  unique  variability)? 

□  Can  be  assessed  using  multiple  regression  analyses 

■  Multiple  metrics  entered  as  predictors  of  a  DV 

■  Each  metric  significant  beta  when  all  considered 
together? 

■  Significant  change  in  R-squared  when  new  metric  added? 
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Incremental  Predictive  Validity 


Sharing  Unique  Information 


Approach  to  the  Study  of  Metrics 


6 

«U4bi»  kr  HaiiMAn  h  indtAg 


I 


ILLINOIS 

uniMf 


EW 


Emerging  Lessons 


□  Experimental/theoretical  constructs  lack  clarity 

□  Primary  focus  on  emergent  states;  lack  of  attention  to 
development  of  team  cognition  or  corresponding 
cognitive  processes 

□  Incremental  validity  often  not  tested 

□  Content  and  method  often  confounded 


□  Over  reliance  on  self  report  and  single  method 

□  Method  (Messmer-Magnus  &  DeChurch,  2007),  and 
indexing  choices  (Smith-Jentsch,  Mathieu,  &  Kraiger, 
2005)  make  a  difference 
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Major  Designs 

□  Meta-analysis  (in-progress) 

□  Predictive  validity  studies  (first  in-progress) 

□  Metric  "bake-off's" 

□  Experimental  tests  of  obtrusiveness 

□  Manipulation  of  task  characteristics 
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Meta-Analysis  of  Prior  Studies 

Measuring 

Components  of  Macrocognition 
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Cognitive  Team-related  Constructs 


CKI  Model 

Schema 

Theory 

Transactive 

Memory 

Mental  Model 
Theory 

Situation 

Awareness 

Theory 

Recognizing 

Expertise 

Teammate 

Schema 

Teammate 
knowledge 
consensus  & 
accuracy 

Mental  models 
of  teammates 

Mental  models 
of  team 
interaction 

Individual 

Knowledge 

Development 

Knowledge 

stock 

Task  & 
Equipment 
mental  models 

Team  Problem 
Model 

Pattern 

Recognition 

Team  Situation 
Awareness 

Quantitative  Database 


Content 

Elicitation 

Aggregation 

Potential  Moderators  & 
Mediators 

Method 

Construct 

Validity 

Knowledge 

Type 

Emergent 

Outcome(s) 
Focused  On 

Method 

Source 

Aggregation  Method 

Task/Team  Properties 

Mathieu  et  al. 
(2000) 

•  Task 
'  Team 

■  Declarative 

■  Shared ness 

■  Pairwise 
ratings 

-  Seif- 
report 

-  UCINET-  QAP 
correlation  (task  had 
two  members) 

■  2  person  teams 

■  Flight  combat  sim. 

■  Novice,  lab 

Espevik  et  al. 
(2006) 

■  Equipment 

■  Team 

■  Team 
interaction 

■  Declarative 

-  IPK 

■  Question¬ 
naire 

■  Self- 
report 

-  N/A 

■  6  person  teams 

■  Tactical  submarine  sim. 

■  Active  duty  officers 

Marks  et  al. 
(2000) 

•  Team 
interaction 

■  Procedural 

■  3-way 
overlap 

■  Pairwise 
ratings 

-  Seif- 
report 

■  Pathfinder  C  index  for 
each  pair  of  team 
members  averaged 

■  3  person  teams 

■  Apache  helicopter  sim. 

*  Novice,  lab 

■  Team 
interaction 

■  Procedural 

■  3-way 
overlap 

■  Concept 
mapping 

-  Seif- 
report 

■  %  of  shared  concepts 
placed  identically  on 
map 

■  3  person  teams 

■  Tank  sim. 

■  Novice,  lab 
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Predicting  Effective  Team  Problem 
Solving  Using  Metrics  of  Macrocognition: 

How  Many  Distinct  Constructs  Exist  and 
What  are  Their  Relations? 
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Relations  Among  Macrocog nitive  Constructs 

□  Initial  experiment  designed  and  approved  by  UCF 
Human  Subjects  Review  Board 

□  Piloted  using  40  participants  this  summer 

□  Scheduled  to  begin  actual  study  in  two  weeks 


Secondary  Task  Environment 
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Experimental  Tasks 


□  Commonalities 

■  Non-combatant  evacuation  scenarios 

■  Require  team  problem  solving  processes 

■  Must  resolve  ambiguity 

■  Product  of  the  team's  performance  is  a  plan 

■  Both  allow  for  face-to-face  or  distributed  communication 

□  Differences 

■  Team  size 

■  Hierarchical  position  in  multi-team  system 

■  Degree  to  which  task  involves  social  cues 

■  Distance  from  the  disturbance 
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Hospital  Emergency  Room  Simulation 


Team  Roles 


Customer  service  personnel 
(participants) 


ER  doctor/nurses 
(live  confederate) 


Remote  Hospital  Personnel 
(simulated  characters) 


Police  officer 
(live  confederate) 


Scripted  Events 
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Shared  and  Unshared  Data 


□  Participants  share  a  common  view  of  ER 
waiting  room 

□  Participants  also  have  access  to 
independent  data  screens 

□  Must  perform  routine  planning  tasks  while 
servicing  customers  prior  to  the  conflict 
event. 
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Problem  Solving  Element 


□  After  interacting  with  video-based  and  live 
confederates,  participants  are  faced  with  a 
hostage  situation 

□  Must  aid  police  in  determining  how  many 
hostages,  who  they  are,  who  is  involved  in  the 
disturbance  and  what  it  is  about. 

□  Clues  are  revealed  as  situation  unfolds 
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Macrocog nitive  Constructs  to  be  Measured 

in  Initial  Study 

□  Team  problem  models 
(a.k.a.  situation  awareness) 

■  Methods:  Paired  comparison  ratings,  card-sorting 

□  Sharing  of  unique  knowledge 
(a.k.a  information  exchange) 

■  Methods:  Eye-tracking,  communication  analysis 
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Relatedness  Ratings 


Participant’s  SME’s  Ratings 
Ratings 


-2  -1 

0 

Strong  negative 

Unrelated 

-2 

)  ^ 

Strong  negative 

Unrelated 

©  ■’ 

© 

Strong  negative 

Unrelated 

Card  Sorting  Task 


Macrocog nitive  Constructs  to  be  Measured 

in  Initial  Study 

□  Team  problem  models 
(a.k.a.  situation  awareness) 

■  Method:  Paired  comparison  ratings,  card-sort 

□  Sharing  of  unique  knowledge 
(a.k.a  information  exchange) 

■  Methods:  Eye-tracking,  communication  analysis 
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Hospital  Rules 


Rule  1;  Patiaits  in  tiie  ER  are  not  seen  by  a  doctor  on  a  fii^t-conie  first  seiz  ed  basis:  Hie  hiage  nin  se 
detennines  theii  ordei  on  the  basis  of  need 

Rule  2;  P.itient  infoiination  is  confidenh.al.  Each  patient  lias  the  liglit  to  decide: 

•  Wlio  nwy  see  liini  or  her 

•  Wlio  nia\’  ha\  e  infoiination  about  their  condition 

•  Wliethei  you  niay  re\  eal  tliat  tlie}-  ai  e  e\'en  registered  as  a  patient  in  the  hosi>ital 

Rule  3;  Docton?  ha\e  the  liglit  to  den\'  access  to  a  patient  if  it  is  in  the  best  interest  of  theufiCaif  s 
medical  conditioii 

Rule  4:  You  will  be  fired  if  you  lie  \'eibally  or  in  wiiting. 

Rule  5;  No  racial  or  sexual  liai-assing  conunents. 


Patient 

Gender 

Consent 

^dical  Condijiim 

Notes 

Name 

Michael 

Rayfield 

Male 

Doctors  dayna^^^ 
visitors 

Severe  trauma 
from  auto  accident 

Currently  in  the  intensive 
care  unit 

Jane  Doe 

Female 

Non-publicity  patient 

Complications 
during  miscarriage 

Real  name:  Kayla 

Johnson 

Carmen 

Diaz 

Female 

Only  grandfather, 
Manny  Diaz,  may 
see  Carmen 

Several  broken 
bones  and 
concussion 

Mother  suspected  of  child 
abuse.  Sire's  not  aibwed 
to  see  the  child. 

Macrocognitive  Constructs  to  be  Measured 

□  Stress 

■  Methods:  Heart  rate,  self-report 

□  Communication  delivery 

■  Methods:  Pitch,  intensity,  Likert-type  scales 


□  Problem  solving  accuracy 

■  Method:  Checklist 


«U4bi»  kr  HaiiMAn  h  indtAg 


ILLINOIS 

uniMf 


Who  is  still  in  the  lobby,  who  is  likely  involved  in  the 

disturbance,  what  is  it  about? 


Pilot  Data  from  Secondary  Task 


□  Physiological  metrics  correlate  with  communication  analysis 

■  Heart  rate  and  blood  pressure  predict  information  sharing 

■  Vocal  intensity  predicts  communication  delivery 

□  Eye  tracking  data  accurate  enough  to  reliably  distinguish 
objects  the  following  distances  apart  from  one  another: 

■  7  inches  at  8  feet  away  (across  the  room) 

■  3  inches  at  2  feet  away  (on  a  monitor) 

□  Eye  tracking  equipment  does  not  bother  participants 
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Comfort  Level  of  Eye-tracking  goggles 
1  (not  at  all)  -  5  (a  great  deal) 


Questions 

Means 

Standard 

Deviations 

To  what  degree  did  the  eye  tracking  equipment  obstruct 
your  view  during  simulation  task? 

1.50 

.548 

To  what  degree  did  the  eye  tracking  equipment  obstruct 
your  view  during  the  card  sort  task? 

1.33 

.516 

To  what  degree  did  you  have  to  adjust  to  the  eye  tracker? 

1.50 

1.225 

To  what  degree  did  the  eye  tracker  caused  discomfort  or 
irritation  to  your  eyes? 

2.00 

1.095 

To  what  degree  did  the  weight  of  the  eye  tracker 
equipment  caused  you  discomfort? 

1.67 

1.211 

To  what  degree  did  you  feel  intimated  having  to  wear  the 
eye  tracking  equipment? 

2.33 

1.633 

To  what  degree  did  the  eye  tracker  equipment  affect  your 
ability  to  use  the  mouse  during  your  tasks? 

1 

0 

To  what  degree  did  the  eye  tracker  equipment  affect  your 
ability  to  use  the  keyboard  during  your  tasks? 

1.50 

.548 

To  what  degree  did  the  eye  tracker  equipment  make  you 
feel  awkward  or  uncomfortable  while  interacting  with  co¬ 
workers,  patients  &  supervisors  in  the  simulation? 

2.00 

1.265 

To  what  degree  did  the  eye  tracker  equipment  distract  your 
overall  performance? 

1.50 

.837 

Overall,  how  uncomfortable  was  it  to  wear  the  eye  tracking 
goggles? 

2.17 

.983 

Summary 

□  Seeking  to  uncover  a  measurable  and 
parsimonious  model  of  macrocognition 

□  Searching  for  converging  results  across 
research  methodologies 

□  Triangulation  of  measurement  strategies 
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